ignite-28305 backpressure to limit in flight partition operations per…#7950
Draft
EgorKuts wants to merge 1 commit intoapache:mainfrom
Draft
ignite-28305 backpressure to limit in flight partition operations per…#7950EgorKuts wants to merge 1 commit intoapache:mainfrom
EgorKuts wants to merge 1 commit intoapache:mainfrom
Conversation
EgorKuts
force-pushed
the
ignite-28305
branch
6 times, most recently
from
393915b to
9140021
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before this change, a node could accept an unbounded number of partition operations, which could lead to the following problems:
Long-queued operations no longer make sense to the client by the time they are executed, forcing the node to perform useless work before it can start handling new requests.
That useless work still consumes node resources (CPU, memory, threads).
Eventually this can lead to OOM.
To prevent such scenarios, this change introduces a node-level semaphore shared across the replica manager and thin-client connector that limits the total number of concurrent partition operations to
maxInFlightPartitionOperationsPerCore * availableProcessors
When the limit is reached, new requests are rejected immediately with ReplicaOverloadedException so clients can back off, while already accepted operations complete uninterrupted.
The limit is disabled by default
(maxInFlightPartitionOperationsPerCore = 0)
Operator can start tuning this around 512 per core and adjust based on observed heap usage and rejection rate under peak load.
https://issues.apache.org/jira/browse/IGNITE-28305